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Uniqueness  and  Eventual  Uniqueness  of  Optimal  Designs  in  Some  Time 

Series  Models,  II 

By  Randall  L.  Eubank,  Patricia  L.  Smith,  and  Philips  W.  Smith 
Southern  Methodist  University  and  Old  Dominion  University 

Short  Title:  Optimal  Designs  in  Time  Series 

Summary.  Earlier  results  on  the  uniqueness  and  eventual  uniqueness  of 
optimal  designs  for  certain  time  series  models  are  extended  to  a  wider 
class  of  processes  which  includes  those  with  covariance  structures  such 
as  that  of  multiple  integrals  of  Brownian  motion  and  Brownian  bridge 
processes.  The  relationship  between  the  problems  of  regression  design 
for  time  series  and  piecewise  polynomial  approximation  with  free  break¬ 
points  is  discussed  and,  consequently,  asymptotic  results  obtained  by 
Sacks  and  Ylvisaker  (1970)  are  seen  to  hold  under  weaker  assumptions 
for  these  processes. 

1.  Introduction.  Consider  the  linear  regression  model  in  which  a 
stochastic  process,  Y,  is  observed  having  the  form 
(1.1)  Y (t)  =  Bf(t)  +  X(t),  t  e  [0,1]  , 

where  3  is  an  unknown  parameter,  f  is  a  known  regression  function  and  X 
is  a  zero  mean  process  with  known  covariance  kernel  R.  The  X  process 
is  assumed  to  admit  exactly  k-1  quadratic  mean  derivatives. 

When  the  Y  process  is  observed,  for  instance,  overall  of  [0,1],  the  re¬ 
producing  kernel  Hilbert  space  (RKHS)  techniques  developed  by  Parzen  (1961a, 
1961b) ,  may  be  utilized  to  obtain  a  linear  unbiased  estimator  of  the  unknown 
parameter  3.  When,  instead,  the  process  is  to  be  sampled  at  only  a  finite 
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number  of  points  the  regression  design  problem  has  been  considered  by 
Sacks  and  Ylvisaker  (1966,  1968,  1970),  Wahba  (1971,  1974),  and  Eubank, 
Smith  and  Smith  (1981a,  1981b).  Under  a  variety  of  assumptions  on  the 
covariance  kernel  R,  and  the  amount  of  information  available  on  the  Y 
process  these  authors  consider  the  problem  of  selecting  an  element,  T, 
from  the  set  of  all  n+1  point  designs 

(1.2)  Dn:  =  { (tQ ,t1#  —  ,tn) 1 0  =  tQ  <  tx  <  •••  <  tn  =  1} 

(where  :*  means  "  is  defined  as")  so  as  to  minimize  the  variance  of  the 
best  linear  unbiased  estimator  (BLUE)  of  8  obtained  by  sampling  according 
to  T. 


Of  particular  importance  for  this  paper  is  the  case  when  the  Y 
process  as  well  as  its  derivatives  may  be  sampled  at  each  of  the  design 
points.  Then,  for  a  given  T  e  D^,  one  may  consider  the  estimation  of 
8  from  the  observation  set 


Y.  :=  {Y(^(t)|t  e  T,  j  «  0, . . .  ,k-l}  . 
K  f  T 


We  denote  by  8V  _  the  BLUE  of  B  based  on  the  observations  Y  .  The 
regression  design  problem,  in  this  setting,  may  be  summarized  as  follows: 
Find  T*  e  D  such  that 


(1.3) 


n 

Var  (8 


k.T*>  '  j"'  V“(8k.T’ 
TED 


n 

In  general  such  a  T*  may  not  be  unique  and  for  this  and  other  reasons  may 
be  quite  difficult  to  construct.  The  computational  difficulties 
associated  with  optimal  designs  led  Sacks  and  Ylvisaker  (1970)  and 
Wahba  (1971,  1974)  to  develop  an  asymptotic  solution  to  problem  (1.3) 


for  several  types  of  covariance  kernels. 

In  a  previous  paper  [5]  we  addressed  the  question  of  uniqueness 


for  optimal  designs*  It  was  found  that  for  covariance  kernels  of  a 
certain  form  and  under  certain  conditions  on  the  regression  function , 
f,  problem  (1.3)  had  a  unique  solution  for  each  n.  In  addition ,  under 
weaker  assumptions  on  f,  the  regression  design  problem  was  shown  to 
eventually  (for  all  n  greater  than  some  finite  nQ)  have  unique  optimal 
solutions.  It  is  the  objective  of  the  present  paper  to  extend  these 
uniqueness  and  eventual  uniqueness  results  to  a  wider  class  of  pro¬ 
cesses.  The  class  of  processes  considered  includes  those  having  co- 
variance  structures  such  as  that  of  multiple  integrals  of  Brownian 
motion  and  Brownian  bridge  processes.  It  is  of  interest  to  note  that 
an  algorithm  for  optimal  design  computation  developed  by  Eubank ,  Smith 
and  Smith  (1981b)  will  also  now  be  applicable  to  this  class  of  processes , 
and,  consequently,  may  be  utilized  along  with  the  unicity  results  of  this 
paper  to  develop  a  computationally  feasible  scheme  for  optimal  design 
construction. 

In  Section  2  we  discuss  the  class  of  processes  to  be  considered 
and  present  our  principal  results.  The  proofs  of  these  findings  are 
provided  in  Section  3.  The  techniques  utilized  in  the  proofs  make  it 
possible  to  draw  certain  conclusions  regarding  the  asymptotic  equivalence 
of  variable  breakpoint  piecewise  polynomial  approximation  and  the  regres¬ 
sion  design  problem  considered  by  Sacks  and  Ylvisaker  (1970) .  This  point 
is  discussed  in  Section  4. 

2.  Results  and  notation,  it  is  well  known  (c.f.  Parzen  (1961a,  1961b)) 
that  the  covariance  kernel  R  generates  a  F2CHS  which  is  isometrically 
isomorphic  to  the  Hilbert  space  spanned  by  the  X  process.  We  denote  this 
RKHS  by  H  (R)  and  its  associated  norm  by  11*11-.-  Under  the  assumption 

K 

that  f  £  H (R) ,  the  regression  design  problem  may  be  reformulated  as  a 
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minimum  norm  approximation  problem.  Let 


(2.1)  R(l'3U,t)  =  3  , 

and  for  T  e  D  set 
n 

(2.2)  T:=  span  {r(0#^  (  %t)  1 1  e  T,  j  =  0,...,k-l}. 

Then/  Sacks  and  Ylvisaker  (1970)  have  shown  that 

var(6k>T)  =  n^fii;2 

where  R^  T  denotes  the  H (R)  orthogonal  projector  for  the  subspace  R^ 


As  I lRk,xfl lR 


|f  -  Tf| |R,  it  follows  that  the  optimal 


design  problem  is  equivalent  to  finding  a  T*.  e  D  such  that 

n 

(2-3)  Ilf-R^fllR-inf  IU-  Vt'H*  • 


In  general,  very  little  is  known  about  the  properties  of  T*  for  finite 
n.  However/  for  the  type  of  processes  studied  in  this  paper  several 
positive  statements  can  be  made. 

We  now  restrict  our  attention  to  a  specific  class  of  processes 
and  their  corresponding  covariance  kernels.  Let 

1  /  x*-1/*.  x*-l 

(2.4)  K(s,t)  =  /  — - du 

(k-l)l 

and  let  2(*)  denote  the  corresponding  normal  process,  i.e.,  a  (k-1)- 
fold  multiple  integral  of  Brownian  motion.  Define  a  new  process,  W,  by 


Z(t)  -  E  [Z  (t)  |z(%),  j  *k-q, . . .  #k-l]  ,  0  <  q  <  k. 


(2.5)  W(t) 


q  -  0 


TV, 


Unless  noted  to  the  contrary,  it  will  be  assumed  in  subsequent  dis¬ 
cussions  that  R  is  given  by 
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(2.6)  R(s,t)  =  Cov(W(s)  ,W(t))  . 

More  specifically,  for  0  <  q  <_  k,  R  cam  be  shown  to  have  the  form 

(2.7)  R(s,t)  -  K(s,t)  -  V(s)'B”1V(t) 
where  V(s)  is  a  vector  whose  ith  component  is 

«•«  v-  -  Eiif1  ft&r-.  - . 

and  B  is  a  qxq  matrix  with  i  jth  element 

(2.9)  b  :  =  K(k"q+i'k"q+j)(l,l)  ,  J1  <!-.«) q~1~1  i,j  =  0 . q-1. 

11  ij  '  j0  (q-i-l)I  (q-j-1)! 

When  q  =  0,  R  is  the  covariance  kernel  corresponding  to  a  (k-l)-fold 
multiple  integral  of  Brownian  motion  whereas  the  case  q=l  corresponds 
to  a  multiple  integral  of  the  Brownian  bridge  process.  The  case  q  =  k 
was  considered  by  Eubank,  Smith  and  Smith  (1981a,  1981b). 

For  processes  with  covariance  kernels  of  the  form  (2.7),  it  is 
possible  to  prove  the  following  theorem  regarding  the  unicity  and 
eventual  unicity  of  optimal  designs  for  the  linear  estimation  of  B. 


Theorem  2.1  .  Let  k  be  a  fixed  positive  integer  and  let  f  e  H(R)OC  [0,1] 
(2k)  (2k) 

with  f  >  0  on  [0,1]  and  log  f v  concave  on  (0,1).  Then,  for  each 
positive  integer  n  there  exists  a  unique  optimal  solution  to  the  regression 

1 2V1 

design  problem  (2.3).  If,  instead,  we  assume  that  f  e  H(R)  with  both  f  >  0 
(2k+3) 

and  f  continuous  on  [0,1]  then  there  is  a  positive  integer  such  that 

for  all  n  larger  than  nQ  problem  (2.3)  has  a  unique  solution. 


In  addition,  it  is  possible  to  prove,  for  general  k,  a  finite  sample 
analog  of  an  asymptotic  result  given  by  Sacks  and  Ylvisaker  (1970)  for  the 
special  case  of  k  *  2. 


1 


•  ♦ 


I* .  < 
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Theorem  2.2.  If  k  is  even  and  f  e  H(R)  with  >  o  then  if  T*  e  D 

- - -  n 

A  A 

is  an  optimal  design  Var (3^.^  T*)  =  Var(Bk  T*). 

Theorem  2.2  has  the  consequence  that  through  the  use  of  optimal 
designs  one  can  obtain  equivalent  resolution  using  (k-l)(n+l)  rather 
than  k(n+l)  observations  for  k  even.  A  result  similar  to  Theorem  2.2 
has  been  given  by  Wahba  (1971)  for  a  certain  class  of  covariance  kernels 
and  specific  types  of  regression  functions. 

The  principal  difficulties  associated  with  the  use  of  optimal 
designs  have  been  their  possible  duplicity  and  computational  infeasi¬ 
bility.  However,  in  some  cases  it  is  possible  to  use  an  algorithm 
developed  by  Chow  (1978)  and  adapted  to  the  regression  design  problem 
by  Eubank,  Smith,  and  Smith  (1981b)  for  optimal  design  computation. 

As  a  result  of  our  proofs  in  the  next  section  it  follows  that  this 
algorithm  is  applicable  for  covariance  kernels  of  the  form  (2.7).  Con¬ 
sequently,  through  use  of  this  algorithm  in  conjunction  with  Theorem 
2.1  the  utilization  of  optimal  regression  designs  may  now  be  considered 
as  a  viable  estimation  technique  for  the  type  of  processes  studied  in 
this  paper.  In  addition,  this  entails  that  Theorem  2.2  has  both  prac¬ 
tical  and  theoretical  implications  for  these  processes. 

3.  Proofs  of  theorems.  In  this  section  we  analyze  the  structure 
of  the  optimal  design  problem  for  covariance  kernels  of  the  form  (2.7). 
In  particular,  we  prove  Theorems  2.1  and  2.2.  First,  however,  it  will 
be  useful  to  consider  certain  recent  results  regarding  piecewise  poly¬ 
nomial  approximation  which  will  be  required  in  the  proofs. 
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For  1  £  q  <  k  denote  the  set  of  polynomials  of  order  q  (degree  <  q) 


by  P^  and  for  a  given  T 


(t  f t  ,  ...,t  )  e  D  let  P  represent  the  set 
0  1  n  n  K  ,t 


of  piecewise  polynomials  on  [0,1]  with  breakpoints  at  t,,...,t  Also, 

-  1  n-l 

denote  by  P  :  =  U  P  the  set  of  all  piecewise  polynomials  of  order  k 
K,n  _  _  1C  *  x 
TeD 


with  n-l  distinct  breakpoints  in  [0,1].  As  most  of  the  work  which  follows 

will  involve  approximation  in  the  L  [0,1]  norm  we  make  the  identifications 

2 

for  f,  g  £  L  [0,1] , 

2 


J1  f(x)g(x)dx 

and 

|  |f  |  |  *  {  J1  f  (x)2dx>ls  . 
0 


2 

Given  f  £  L  [0,1]  it  is  frequently  of  interest  to  consider  finding 
a  best  approximation  to  f  from  P^  n-  Thus,  one  attempts  to  find  a  T*  £ 
such  that 


(3.1)  || f  -  PkfT*f||  =  inf | | f  -  Pk  Tf|| 


TED 


k,T 


where  P  denotes  the  L-[0,1]  orthogonal  projector  for  P  .  Several 
K ,  T  —  K ,  T 

recent  results  due  to  Barrow,  et  al.  (1978),  Barrow  and  Smith  (1978)  and 

Chow  (1978)  are  available  regarding  the  properties  of  T*.  These  results 

n 

were  discussed  in  Eubank,  Smith  and  Smith  (1981)  and  are  stated  again  here 
for  completeness. 


k 

Theorem  3.1.  Let  f  €  C  [0,1]  with  f  >  0  on  [0,1]  and  log  f  concave 

on  (0,1).  Then,  for  each  positive  integer  h,  T*  in  (3.1)  is  unique.  If, 

n 

k+3  (\c) 

instead,  it  is  assumed  that  f  e  C  [0,1]  with  f  >  0  on  [0,1]  then  there 

* 

is  a  particular  integer,  n^,  such  that  for  all  n  >nQ  is  unique. 


8 


(k) 

Proposition,  Let  f  >0  and  let  p*  be  a  best  L^tO^l]  approximation  to  f 

from  P  .  Then,  if  k  is  even,  p*  e  C[0,1]. 
k  ,  n 

We  now  consider  the  structure  of  the  optimal  design  problem  for  co- 

variance  kernels  of  the  form  (2.7).  For  fixed  values  of  k  and  q,  H(R)  is 

seen  to  consist  of  functions  with  k-1  absolutely  continuous  derivatives  and 
is  given  by 


(3.2)  H(R) 


{f |f(j) (0)=0,j=0,...,k-l,f(j) (l)=0f j=k-q,...,k-l,f(k)el2  [0,1] } ,l<q<k, 
(0)=0,  j  =  0,...,k-l,f°°  e  I^[0,1]},  q=0. 


where  the  norm  for  f  e  H(R)  is 


(3.3) 


(k  ) 

Now  let  r  .  (s)  -  R  (s,t)  ,  j  =  <),.•♦, k-1,  and  define  the  corresponding 
t  #  3 

(L2[0,1] Jsubspace 

(3.4)  r£  T:=  span{R^k'^ (• ,t) |teT, j  =  0,...,k-l} 

=  span{r  .  |teT,  j  =  0,...,k-l} 

Then,  in  view  of  (3*3)  ,  we  have 

(3.5)  I  |f  -  Tf  I  |R  =  I  |f  (lc)-(Rlc #Tf) (k)  1 1  =  I  |f(k)-  ^,Tf(k)  1 1 

k  k 

where  T  is  the  1^10,1]  orthogonal  projector  for  T*  Consequently, 

the  optimal  design  problem  is  equivalent  to  finding  the  best  L2[0,1] 

(k)  k 

approximation  to  f  from  the  functions  in  R^ 

The  preceding  discussion  serves  to  motivate  further  study  of  the 
k 

form  of  R^  T-  Therefore,  let  K(s,t),  v^(s)  and  b^  be  as  given  in  Section  2 

and  for  0  <.  j  £  k  let 

V<*>(8)  *  (v<jU),...,v<jJ(s))*. 

u  q-l 


(3.6) 
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Then,  the  elements  of  T  are 


(3.7)  r  .(s)  =  K(k,j)(s,t)  -  v(k)(s)'lfVj)(t),  0  <  j  <  k  -  1  . 


Now  set 

(3.8)  p*(s) 


i-1 

(t-s)+ 

(i-l)I 


1  =  1  g  .  m  •  ,  1C  , 


with 

n 

(3.9)  p1  (s)  s=  pj(s)  =  -J~y, 


and  note  that  this  implies  that 

(3.10)  Kk'^(s,t)  =  p^  3(g)  9  j  -  o, . . .  ,k-l , 


and 

(3.11)  (s)  =  * (s)  ,  i  =  0, . . . ,q-l  . 

Upon  observing  that 

(3.12)  V?  (t)  ==  ^p^  ^  t  i  *  1,  j  —  0,  •  •  •  ,k— 1 

and 

(3.13)  b..  -  <pq‘\  p’-3)  ,  i.j  -  0 . q-1. 

it  is  readily  seen  that  r  .  is  the  error  function  resulting  from  the 

t ,  j 

I^fOfl]  approximation  of  p^  J  from  P^.  Using  P^  to  denote  the  orthogonal 

projector  for  P  ,  the  basis  elements  for  R  are  succinctly  summarized  in 
q  K  ,T 

the  following  array. 


*  «  -  - 
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The  zeros  in  (3.14)  which  occur  at  tg  =  0  and  t  ■  1  arise  from  the  fact 

j  ]^- j 

that  p  (s)  vanishes  identically  and  that  p  e  P  for  j  =  k-q,...,k-l. 

o  q 

This  may  be  interpreted  as  indicating  that  observations  taken  on  the 

corresponding  derivatives  of  the  Y  process  at  these  points  will  provide 

no  information  about  B.  This,  of  course,  is  otherwise  clear  from  the 

boundary  conditions  that  functions  in  H(R)  must  satisfy. 

k 

It  follows  from  (3.14)  that,  for  T  e  D  t  R  CP.  Thus,  in  view 

n  jc  ,  t  k  ,  r 

of  (3.5),  it  is  now  apparent  that  the  problem  of  optimal  design  unicity 

is  similar  to  the  problem  addressed  by  Theorem  3.1.  The  only  obstacle 

k 

preventing  the  immediate  proof  of  Theorem  2.1  is  that  T  is  properly 

contained  in  P  .  To  resolve  this  difficulty  we  prove  the  following 
k ,  T 

lemma. 


Lemma. 


Let  f  e  H(R)  and  T  e  D  . 

n 


*(k)  -  pk  (k) 

f  from  P,  _  is  fc  f 
k,T  k,T 


Then  the  best  L2[0,1]  approximation  to 


Proof.  From  (3.14),  we  may  write 


Using  the  boundary  conditions  in  (3.2)  and  integration  by  parts  it  can  also 
be  shown  that  for  any  g  e  H(R)  and  p  e  P^  , 

/  g^k)  (x)p(x)dx  =  0  . 

0 

(k)  k  (k) 

In  particular,  as  f  -  R  f  is  in  H (R) ,  this  entails  that  f  -  f 

k  (k)  «k  (k) 

is  orthogonal  in  L2  to  both  R^  T  and  P^.  Hence  f  Tf  is  orthogonal 

to  P  and  the  lemma  is  proved. 

K ,  T 

Theorem  2.1  now  follows  directly,  through  use  of  the  lemma,  from 

(k) 

Theorem  3.1  upon  replacing  f  by  f  .  To  obtain  Theorem  2.2  we  use  the 
Proposition  to  see  that  when  k  is  even  and  T*  is  an  optimal  design 
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Rk,T*f<k'  E  ^,T*nC[°'1  1  =  E  T*'  j  *  °'"-'k-2>  * 


4.  Piecewise  polynomials  and  regression  design.  Sacks  and  Ylvisaker 
(1970)  have  considered  the  asymptotic  behaviour  of  optimal  regression 
designs  for  a  particular  class  of  covariance  kernels.  Under  the  assump¬ 
tion  that  Q  is  a  covariance  kernel  satisfying  certain  assignations  (see 
Assumptions  A,  B  and  C  in  Section  2  of  Sacks  and  Ylvisaker  (1968))  R  is 


assumed  to  have  the  form  ,  .  ,  „ 

k-1  k-1 

fl  fl  (s-u)  +  (t-v)+ 

(k-l) !2 

0  0 


(4.D  R(s,t)  =  j1/ 


Q(u,v)dudv 


It  is  then  shown  that  for  this  type  of  covariance  kernel  when  f  admits 

the  representation 
1 

(4.2)  f(t)  =  /  *(s)R(s,t)  ds 
for  some  (j)  e  C[0,1]  then 

-v.  2  „  , .  2  1  „  ,  2k+l 

(4.3)  limn  inf  |  |  f  -  1 1  =  (2k)t(2k+l)!  lV(x)  +  dX  ^ 


n-^° 


TeD 


The  technique  utilized  in  the  proof  of  this  result  was  to  first  show  that 

(4.3)  held  in  the  special  case  when  Q  has  the  form  QQ(srt)  *  mints, t)  and 
then  extend  this  outcome  to  general  Q  through  the  use  of  a  mapping  from 
H(Q)  onto  H(Qq) . 

Results  similar  to  (4.3)  have  been  obtained  by  several  authors  for 

the  asymptotic  behaviour  of  the  error  from  the  best  L2[0,1]  approximation 

of  a  function  from  P.  •  For  instance,  Burchard  and  Hale  (1975)  have 

Jc ,  n 

shown  that  if  f  e  L^[0,1],  where 

(4.4)  l£[0,l]  =  {f  |  f^absolutely  continuous,  j=0, . . .  ,k-l,/  |  f  ^\x)|dx  <“} , 

1  f\ 


then 


9V  2  k»  (  1  2/2k+l  j  2k+l 

(4“5)  TeD^f  "  "  (2k)  !  (2k+l) !  j^0(f  (x) }  dx  | 

n  } 

We  now  indicate  the  connection  between  these  two  bodies  of  theory. 

The  case  of  Q(s,t)  =*  min(s,t)  corresponds  to  the  event  when  R  is 

k  (2k) 

given  by  (2,4)  and  $  =  (-1)  f  .  As  this  entails  q  =  0  in  our  work  it 

k 

now  follows  from  (3.14)  that  in  this  case  R,  =  P  and,  therefore,  in 

iC  ,  x  JC  ,  T 

this  instance,  the  optimal  design  problem  corresponds  precisely  to  the 

(k) 

problem  of  best  approximating  f  from  n.  This  clearly  indicates  the 

reasons  underlying  the  similarity  between  the  asymptotic  behaviour  of  best 

piecewise  polynomial  approximations  and  optimal  designs  (for  other  remarks 

on  this  relationship  see,  e.g.  McClure  (1975)).  We  now  utilize  the  duality 

between  these  two  areas  of  study  and  note  that,  for  the  processes  considered 

in  this  paper,  the  work  of  Burchard  and  Hale  (1975)  has  the  consequence 

(k)  k 

that  (4.3)  still  holds  under  the  weaker  assumption  that  f  e 
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